Accuracy Issues for Automated and Artificial Intelligent Residential Valuation Systems

This paper extends current work in the field of Valuation Accuracy and Valuation Variance. These issues are examined in respect of Automated and Artificial Intelligent Valuation Systems. The issue of how accurate an automated system must be to compete with manual valuations is explored. The paper relies upon existing literature, a survey of practising valuers’ and a previous valuation variation study for residential properties in Adelaide. The results show that valuers expectations of valuation accuracy greatly exceed the actual results achieved and that their expectations for automated systems are unrealistic on this basis. Introduction The development of Artificial Intelligent Systems for the valuation of residential property is occurring rapidly. In South Australia, such a system is under development at the University of South Australia. One of the difficulties is finding an accuracy level to test this system against. Most valuers’ see even small error terms as unacceptable. This is based on the premise that their manual valuations are “always right”. This paper explores recent literature that seriously questions the accuracy and variations of valuations. A valuation variation study in Adelaide is used to show that a wide range of values is typical even for basic residential valuations. A small survey of practising valuers is used to gauge opinions of the accuracy of manual valuations compared to automated systems. It is hoped this paper with help fill a gap in the literature in this area. In particular the issues of accuracy in automated systems. This will become a major issue as more automated and artificial intelligent systems are developed in Australia and Internationally. Accuracy Issues for Automated and Artificial Intelligent Residential Valuation Systems – Rossini Page 2 Automated and Intelligent Systems Technological changes are leading to rapid changes in the way that professionals perform their business. Professionals in the property industry are no exception. Baen et al (1997) discusses a range of affects to property professionals which are emerging in the US which they believe may lead to very significant reductions in employment levels. One of the technological changes is the introduction of automated and intelligent valuation systems. These are most often used in basic residential valuations, particularly for valuations to support finance. The foundation of these systems was in the mass appraisal field. Early computer aided systems became popular in the USA in the early 1980’s. A practical discussion of the methods by Sauter, B. W. (1985) suggested a logical approach to automate the typical actions of the valuer. Kershaw (1997) used the general approach suggested in developing the current basic prototype at the University of South Australia. Other methodologies have been suggested by Jensen (1984) who suggests a variety of different approaches. Rossini et al (1992, 1993) suggested the first system that included automated valuation concepts based in an Australian setting. This proposed system included a complex sales history management system based on government databases, automated time series system and automated valuation system. The first stage of the system (data management) was released commercially in 1994 (Kershaw, 1994, 1996) with a tested prototype of the automated time series being presented in January 1997 (Kershaw et al 1997). Initial work on a prototype automated valuation system was demonstrated in late 1997 (Kershaw, 1997) There is evidence that several systems are commercially available in the USA. Systems in Marion County, Linn County and Benton County, Oregon are described by Detweiler (1996). These are working systems (Computer Assisted Real Estate Appraisal System, CAREAS) used by a variety of users but primarily for mortgage finance purposes. Several Internet sites refer to similar systems. Jensen (1990) reports on the initial development of a system for Seville in Spain. It is clear that systems vary considerably in each location, primarily due to differences in purchaser preference (market price allocation) as well as variations in the type, amount and quality of data that is available. The current systems are based on relatively straightforward statistical modeling systems with an expert system element. More complex systems have also been proposed but there is little evidence of any being applied in at this stage. Eckert et al (1993) proposed a more complex system using econometric modeling. They stated that a Computer-Assisted Real Estate Appraisal (CARA) would have wide application, particularly in the risk management of mortgage loan portfolios. “First, the model can be used to provide an automated, market-based valuation prior to an initial onsite inspection. CARA’s most important risk management contribution, however, is its ability to provide an automated review appraisal based on the comparison properties cited in a subject appraisal as well as other subject and comparison properties. Finally, CARA can automatically update original sale prices to current market levels”. The latest major change has been the suggestion that systems may use true artificial intelligence rather than an automated approach. Such systems would normally be based on artificial neural networks. The distinction between automated systems and artificial intelligent systems is clarified by Rayburn (1995). While the automated systems use statistical techniques such as regression, intelligent systems based on neural networks are capable of learning in a more complex and non-linear manner. The opportunity for their use has been investigated in recent years. For example Borst (1991) reported the use of ANN to data sets of family residences in New England. Tay and Ho (1992, 1994) examined sets in Singapore using 833 residential apartment properties for training and tested this against 222 case set of similar apartment properties. Do and Grudnitiski (1992) used data from a multiple listing service in California while Evans (1993) worked with residential housing in the United Kingdom. Recent works comes from Worzala (1995), Borst (1995, 1996), McCluskey (1996a, 1996b) and Rossini (1997a, 1997b, 1997c). Rossini’s research was based on data from South Australia and demonstrated that the results from artificial neural networks could potentially produce superior results to more traditional econometric models in certain circumstances. What is not clear from the research is the level of accuracy that is expected. This paper will examine various aspects of valuation accuracy and variations and draw some conclusions about what is required in automated or intelligent systems. Accuracy Issues for Automated and Artificial Intelligent Residential Valuation Systems – Rossini Page 3 Valuation Accuracy and Valuation Variation Before it is possible to assess the accuracy needed from an automated or intelligent valuation system, it is necessary to consider the broader question of valuation accuracy. The issues of accuracy has always been of interest to valuers and their clients but only in recent years has there been an active research focus on this issue. In the UK, Hager & Lord (1985) started the modern debate and quoted some accuracy figures. This was followed in the UK by Hutchison (1996) and Crosby et al (1998a, 1998b). Newell & Kishore (1998a, 1998b) and Parker (1998) introduced some accuracy and variation estimates from Australia. Amongst these papers were many others that discussed methodology. (Lizieri et al, 1991 & 1993;Brown, 1992 & 1998;McAlister, 1995; Wiltshaw, 1996; Boronico, 1997). There is a clear split between the testing of valuation variance and valuation accuracy. This seems to have been a point of some confusion in some of the research. Lizieri et al describe these as “The former (Valuation Variance) occurs where two or more valuers arrive at substantially different values for the same property.......it is difficult to statistically test the significance of observed variations. The latter (Valuation Accuracy) concerns the relationship between achieved prices and prior valuations.” Or as Crosby et al (1998b) succinctly puts it “...valuation accuracy (valuations against price) and valuation variance (valuations against valuations)” There is also considerable debate about the analytical methods used in some of the research that may tend to over estimate the errors. This is a particular argument of Brown et al (1998) who argues that bootstrapping is a more appropriate statistical method given sample sizes used in most of the research. Notwithstanding this there is some evidence of the accuracy of valuations from the available research. The results from Hager & Lord (1985), Hutchison (1996), Morgan, Drivers Jonas/IPD, Matysiak & Wang, Blundell & Ward, all as quoted in Crosby (1998a), Newell (1998a), Parker (1998) and DeVries (1992) are summarised in Table 1. Table 1 Summary of Valuation Variance and Valuation Accuracy Research Hager & Lord(1985) Hutchison (1996) Morgan(1993)* Newell(1998a) Driver Jonas/IPD* Matyiak & Wang* Blundell & Ward* Parker (1998) DeVries (1992) % Absolute Error Office Shops Reversion Rack Rent Commercial Commercial Commercial <5% 40 50 n/a n/a n/a n/a n/a < 10% 90 80 69 61 30 35 n/a < 15% n/a n/a n/a n/a 55 n/a n/a < 20% 100 90 93.5 85 70 80 n/a Mean Absolute Error n/a n/a 9.53% 10.50% n/a n/a 10.78 * As quoted in Crosby (1998a) ** Typical Figure papers quotes figures over a four year period Percentage of properties that fall into each Accuracy Range